Introduction to This Special Issue on Multimodal Interfaces

The growing emphasis on multimodal interface design is fundamentally inspired by the aim to support natural, flexible, efficient, and powerfully expressive means of human-computer interaction that are easy to learn and use. Multimodal interfaces represent a new direction for computing that draws from the myriad input and output technologies becoming available, and that potentially can integrate complementary modalities to yield more synergistic blends than has been possible previously. It is a research direction that seeks guidance from cognitive science expertise on the coordinated human perception and production of naturally co-occurring modalities (e.g., speech, gesture, gaze, and facial movements) during interaction with people and computers. In fact, the realization of successful multimodal systems is dependent on and only can flourish through extensive interdisciplinary cooperation, as well as teamwork among those representing expertise in the individual component technologies. In this special issue, new work on next-generation multimodal interfaces includes articles based on PhD theses by Elizabeth Mynatt of Xerox Palo Alto Research Center, Robert Stevens of the University of York, and Yi Han of the University of Melbourne. It also includes articles based on extensive work by Steve Roth's group at Carnegie Mellon University and from Sharon Oviatt's laboratory at the Oregon Graduate Institute of Science and Technology. Reading these articles, it becomes clear that multimodal interface in no sense refers to a unified and specialized subarea of research. Rather, it is a collection of emerging research areas, like early evening stars gradually glowing more brightly. Some of the clusters already visible include multimodal interfaces that support computing for special populations (e.g., Mynatt, this issue; Stevens, Edwards, & Harling, this issue), research on coordinated multimodal input (e.g., Oviatt, this issue), and work on presentation planning and coordinated multimodal output (e.g., Han & Zukerman, this issue; Roth, Chuah, Kerpedjiev, Kolojejchick, & Lucas, this issue). In addition, new methods for generating research and system development are becoming available, including multimodal simulation techniques for collecting data (e.g., Oviatt, this issue) and agent architectures for implementing integrated multimodal systems (e.g., Han & Zukerman, this issue). Multimodal interfaces have the potential to expand computing to encompass more challenging applications, for use by a broader spectrum of the population, and during more adverse usage conditions. It is not coincidental, then, that most of the articles in this issue present work in support of challenging applications; these include algebra instruction (Stevens et al., this issue), presentation of data summaries (Han & Zukerman, this issue; Roth et al., this issue), and interaction with complex spatial displays such as maps (Oviatt, this issue; Roth et al., this issue). In addition, two articles